Mixed reality measurement with peripheral tool

ABSTRACT

The techniques introduced here enable a display system, such as an HMD device, to generate and display to a user a holographic structure matching a real-world structure. In some embodiments vertices, edges and planes of the holographic schematics are generated via the use of a peripheral tool that is positioned by a user. In other embodiments, other user input indicates the bounds of the holographic schematic. In response to user action, a holographic schematic is made to appear including corresponding real-world size measurements. The corresponding measurements are used to develop a holographic structure that integrates with the holographic schematic.

CLAIM OF PRIORITY

This application is a divisional application of U.S. patent applicationSer. No. 15/595,857 filed on May 15, 2017, titled “Mixed RealityMeasurement with Peripheral Device,” which claims the benefit of U.S.Provisional Patent Application No. 62/481,047 filed on Apr. 3, 2017,titled “Mixed Reality Measurement With Peripheral Device”, and U.S.Provisional Patent Application No. 62/481,052 filed on Apr. 3, 2017,titled “Mixed Reality Generation of Schematics” all of which areincorporated by reference herein in their entirety.

BACKGROUND

Virtual reality (VR) and augmented reality (AR) visualization systemsare starting to enter the mainstream consumer electronics marketplace.AR Head-Mounted Display (HMD) devices are one promising use of suchtechnology. These devices may include transparent display elements thatenable a user to see virtual content transposed over the user's view ofthe real-world. Virtual content that appears to be superimposed over theuser's real-world view is commonly referred to as AR content. DisplayedAR objects are often referred to as “holographic” objects. VR and ARvisualization systems can provide users with entertaining or useful,immersive three-dimensional (3D) virtual environments in which they canvisually (and sometimes audibly) experience things they might notnormally experience in real life.

SUMMARY

The techniques introduced here enable a display system, such as an HMDdevice, to generate and display to a user a holographic schematicdiagram (or simply “schematic”) matching a real-world structure. Aperipheral tool is used to aid with the precision and speed of thegeneration of the holographic schematic. The peripheral tool is capableof multiple physical orientations to indicate a variety of input and istracked by the HMD device using a set of fixed fiducial markers.Software (running on the HMD device) uses a mounted, calibrated cameraon the HMD device to capture the state of the markers and runs aperspective and point (PnP) process to accurately localize the tool.Using the known physical geometry of the tool, individual points anddirections on the physical tool are easily computed.

The HMD device can further be configured to generate and display aholographic structure that appears to be affixed or coupled to thereal-world structure as if the holographic structure was a real-worldstructure. The type of holographic structure may be based upon the typeof real-world structure detected by the HMD device either through userinput or through image recognition algorithms.

Other aspects of the technique will be apparent from the accompanyingfigures and detailed description.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present disclosure are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements.

FIG. 1 illustrates an example of an environment including an HMD device.

FIG. 2 shows a perspective view of an example of an HMD device.

FIG. 3 illustrates a number of examples of peripheral devices that canbe used in conjunction with an HMD device.

FIG. 4 is a flowchart illustrating the operation of a peripheral tool incombination with an HMD device.

FIG. 5 is a flowchart illustrating the holographic generation of amanufactured device fitting a digital schematic.

FIG. 6 is a flowchart illustrating the operation of an HMD devicehandling user input without a peripheral to measure real-worldstructures.

FIG. 7 is an image of a holographic schematic overlaid on a real-worldstructure.

FIG. 8 is an image of a holographic structure generated based on aholographic schematic.

FIG. 9 is an image of an alternate holographic structure generated basedon a holographic schematic.

FIG. 10A is an image of a holographic user interface for selecting styleoptions on a holographic structure where the holographic structure is ina first style configuration.

FIG. 10B is an image of a holographic in a second style configurationafter receiving a selection in the holographic user interface.

FIG. 10C is another alternate holographic structure generated based on aholographic schematic.

FIG. 11 is a block diagram showing an example of various functionalcomponents of an HMD device.

DETAILED DESCRIPTION

In this description, references to “an embodiment,” “one embodiment” orthe like, mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodiment ofthe technique introduced here. Occurrences of such phrases in thisspecification do not necessarily all refer to the same embodiment. Onthe other hand, the embodiments referred to also are not necessarilymutually exclusive.

These and other techniques are described further below. First, however,it is useful to describe an example of an environment and a device inwhich these techniques can be implemented.

FIG. 1 shows an example of an environment including an HMD device 1 thatcan implement the techniques introduced here. In the illustratedexample, the HMD device 1 is configured to communicate data to and froman external processing device 2 through a connection 3, which can be awired connection, a wireless connection, or a combination thereof. Inother use cases, however, the HMD device 1 may operate as a standalonedevice. The connection 3 can be configured to carry any kind of data,such as image data (e.g., still images and/or full-motion video,including 2D and 3D images), audio data (including voice), multimedia,and/or any other type(s) of data. The processing device 2 may be, forexample, a game console, personal computer, tablet computer, smartphone,or other type of processing device. The connection 3 can be, forexample, a universal serial bus (USB) connection, Wi-Fi connection,Bluetooth or Bluetooth Low Energy (BLE) connection, Ethernet connection,cable connection, DSL connection, cellular connection (e.g., 3G, LTE/4Gor 5G), or the like, or a combination thereof. Additionally, theprocessing device 2 may communicate with one or more other processingsystems 5 via a network 4, which may be or include, for example, a localarea network (LAN), a wide area network (WAN), an intranet, ametropolitan area network (MAN), the global Internet, or a combinationthereof.

FIG. 2 shows a perspective view of an example of an HMD device that canimplement the techniques introduced here. The HMD device 20 can be anembodiment of HMD device 1 in FIG. 1. HMD device 20 includes a headfitting, namely, a headband 21, by which the HMD device 20 can be wornon the user's head. Attached to the headband 21 (directly or indirectly)is a transparent protective visor 22 that encloses one or moretransparent AR display devices 23, each of which can overlay holographicimages on the user's view of his real-world environment, for one or botheyes (e.g., by projecting light into the user's eyes). The protectivevisor 22 also encloses various circuitry (not shown) and sensors.

The HMD device 20 further includes one or more eye-tracking cameras 24,one or more microphones 25 to input speech from the user (e.g., for usein recognizing voice commands and providing audio effects); one or moreaudio speakers 26 to output sound to the user; one or morevisible-spectrum tracking cameras 27 for use in capturing images ofsurrounding surfaces to allow tracking of the user's head position andorientation in real-world space and hand gesture recognition; one ormore infrared (IR) spectrum depth cameras 28 for use in determiningdistances to nearby surfaces (e.g., for use in surface reconstruction tomodel the user's environment); one or more IR illumination sources 29for use with the depth camera(s) 28; and one or more visible spectrumvideo cameras 30 for use in capturing standard video of what the usersees. The HMD device 20 also includes circuitry (not shown), which maybe contained within the visor 22, to control at least some of theaforementioned elements and perform associated data processing functions(e.g., speech and gesture recognition and display generation). Thecircuitry may include, for example, one or more processors and one ormore memories. Note that in other embodiments the aforementionedcomponents may be located in different locations on the HMD device 20.Additionally, some embodiments may omit some of the aforementionedcomponents and/or may include additional components not mentioned above.

An HMD device can be used for high efficiency capture andpre-visualization of measurements and projecting final manufacturingresults. Schematic capture of a space typically involves twocomponents—localization and labeling. This solution can improve onexisting techniques in both these regards.

FIG. 3 illustrates a number of examples of peripheral devices, i.e.,hand tools that can be used by a user of an HMD device. A peripheraltool 30 includes a number of fiducial markers 32. Each of the fiducialmarkers 32 is unique such that the HMD device 20 is able to identify theorientation of the peripheral tool 30 from recognition of relatively fewof the fiducial markers 32 (e.g., 5). The size specifications of theperipheral tool 30 are included in the memory of the HMD device 20. Theknown size specifications enable precise identification of a givenlocation or surface by recognition of some or all of the fiducialmarkers on the peripheral tool. The peripheral tool 30 may optionallyinclude a handle 34 to for additional ease of positioning by a user.

The peripherals tool 30 each include a corner or point 36. The point 36is used to identify particular location to generate a particular vertexin the digital space. When the point 36 is placed on a real-worldlocation, the HMD device 20 includes programming to generate a digitalvertex (i.e., a vertex of a holographic image) at that location. Theperipheral tools 30 further include various edges 38. An edge 38 can beused to identify a particular plane or line. When a user positions theedge 38 against a real-world surface, the HMD device 20 is programmed togenerate a digital plane or line of a holographic image relative to theposition of the peripheral tool 30. The parameters of the plane or lineare determined based on HMD device depth finding that identifies thereal-world first deviation/variation from the line or plane as thebounds of the indicated line or plane.

In some embodiments, the point 36 and the edge 38 are used inconjunction where the point 36 indicates one bound of the line or plane.In some embodiments, rather than using the edge 38, the orientation ofthe surface including the fiducial markers 32 is used to determine theplane of surfaces defined by vertices indicated by the point 36.

The orientation of the peripheral tool 30 is aided by positional data ofthe HMD device 20. The HMD device 20 includes sensors in order to be“gravity-aware” to determine the down direction. Data indicative of thedown direction further enables the HMD device 20 to determine theorientation of the peripheral tool 30.

FIG. 4 is a flowchart illustrating the operation of a peripheral tool incombination with an HMD device. In step 401, the HMD device initializesa measurement program. In this step, the HMD device checks orientationand scans local real-world surfaces. In step 402, The HMD deviceidentifies a location of a peripheral tool. The HMD device usesrecognition of fiducial markers to find the peripheral tool in the HMDdevice's field of view.

In step 403, the HMD device identifies the orientation of the peripheraltool. This is performed via identifying the comparative distance of eachof the recognized fiducial markers. In step 404, the HMD devicedetermines based on the orientation of the peripheral tool how to handlethe user input of the peripheral tool location/positioning. The HMDdevice determines whether the user input is requesting a point or anedge/surface. To do this the HMD device interprets the orientation ofthe peripheral tool. In some embodiments, when only a point or corner ofa peripheral tool contacts a surface, the HMD device handles the inputas requesting a point/vertex. When an edge of the peripheral toolcontacts a surface, the HMD device handles the input as requesting aline or edge. When the peripheral tool is laid flat on a surface, theHMD device interprets the input as requesting a plane. Where the userinput indicates a point, in step 405, the HMD device generates a vertexin digital space. Where the user input indicates a plane or a line, instep 406, the HMD device generates a digital plane oriented with theperipheral tool, or a line parallel with the edge of the peripheraltool.

In step 407, the HMD device determines whether or not there areadditional points to capture based on user input (e.g., voice command,button press, gesture, hiding the peripheral tool, or other known userinput known in the art). Where there are additional inputs, the methodreturns to step 402. In step 408, where there are no additional surfacesor vertices, the HMD device generates a holographic schematic of thereal-world structure as indicated/outlined by the peripheral tool.

FIG. 5 is a flowchart illustrating the generation of a holographic imageof a manufactured (or to be manufactured) device corresponding to adigital schematic diagram (“schematic”). In step 501, the HMD deviceinitializes a measurement program. In this step, the HMD device checksorientation and scans local real-world surfaces. This step proceedssimilarly as step 401 in FIG. 4. In step 502, the HMD device generates aseries of vertices for a holographic structure that matches a real-worldstructure. Each of the vertices is identified via user input.

In step 503, the HMD device identifies the holographic structure. Thisidentification may be based on user input or based on automatic imagerecognitions algorithms. The recognition algorithms can use machinelearning to compare shapes to expected models or parameters. Once theHMD device determines a type of structure through recognitionalgorithms, the user is enabled to reclassify the structure type viauser input. In step 504, based on the determination of the character ofthe holographic structure the HMD device generates a holographicmanufactured article to fit the holographic structure positioneddigitally relative to the holographic structure. Examples ofmanufactured articles that can be positioned according to holographicstructures include blinds in window frames, a treehouse in a tree,shelves/cabinets on a wall, a light fixture in a room, and a motorizedchair in a stairwell.

FIG. 6 is a flowchart illustrating the operation of an HMD devicehandling user input without a peripheral tool to measure real-worldstructures. The process of FIG. 6 flows similarly to that of FIG. 4.However, step 602 merely includes user input as opposed to the use of aperipheral tool. An example of suitable user input is pointing with auser's hand. Positioning of hands can substitute as peripheral toolorientation data. In some embodiments, voice commands supplement handgestures to determine the manner in which the HMD device generates theholographic schematic (step 607). Examples of hand position input arethe user tracing a location with their finger, and laying their handflat on a surface and positioning a fingertip at a location.

In some embodiments, a user may merely identify a general vicinity, andthe HMD device, using depth finding, generates a schematic of allreal-world structures located within the general vicinity. While the HMDdevice is operational, it can use its depth camera(s) and associatedprocessor(s) to construct a 3D mesh model of all surfaces in the user'svicinity (e.g., within several meters), or at least of all nearbysurfaces within the user's field of view, including their distances fromthe user (i.e., from the HMD device). Techniques for generating a 3Dmesh model of nearby surfaces by using depth detection (e.g., time offlight) are known in the art and need not be described herein.

In at least one such an embodiment, the user input is merely providingthe HMD an image frame that includes the desired real-world structure.In some cases, the real-world structure covers more than a single imageframe. In such cases, the user pans the HMD or moves thereby generatinga number of consecutive image frames. For example, a user capturing astaircase may indicate a beginning frame to the HMD device, and thenproceed to walk up the staircase while focusing on the stairs. While theuser moves the HMD device captures a schematic of the staircase.

FIG. 7 is an image of at least a portion of a holographic schematic 39overlaid on a real-world structure. In this case, the real-worldstructure is a staircase. Overlaid on top of the staircase is aholographic schematic including a number of virtual vertices 40, virtualedges 42 connecting virtual vertices 40, and virtual surfaces 44 boundedby the virtual edges 42. Other examples of real-world structures thatthe HMD may generate holographic schematics of include a room, a naturalstructure such as a tree or a cave, a car, or other suitable physicalstructure that has visually ascertainable dimensions.

FIG. 8 is an image of a holographic structure 46 generated based on aholographic schematic. In this example, the holographic structure 46 isa set of cabinets positioned in the corner of a room. The HMD 20 devicehas generated a schematic of the room and has created a display of thevirtual cabinets within that room. The HMD device 20 may generate theholographic structure 46 using specifications of correspondingreal-world components. The HMD device 20 may also use a custom shapegenerated based on the particular character and shape of the holographicschematic 39. This enables a user to visualize the look of amanufactured article to install with the real-world structure.

In this example, the HMD device 20 has access to a virtual schematic ofthe corner of a room. The HMD device 20 or another source may havegenerated the schematic. The corner has twelve feet by four feet offloor space (this may not be the entire size of the room, but is theextent of the schematic). Further, the corner is eight feet tall. Thus,the HMD device 20 is able to determine that the corner will accommodatea ten feet by one and a half feet cabinet that is three and a half feettall, and a counter having a similar footprint that is three feet tall.There is one and a half feet of remaining counter space. Thus, the HMDdevice 20 displays a holographic structure 46 (a cabinet) that matchesthis profile.

A number of real-world structures may be viewed with an HMD device insimilar manner as described above, so as to appear to have holographicstructures applied to them. For example, a real-world tree may includeholographic insertion of a treehouse. A real-world car may include aholographic spoiler. A real-world house may include a holographicbalcony.

FIG. 9 is an image of an alternate holographic structure generated basedon a holographic schematic. In this example, the holographic structure46 is a rail for an automated stairway chair. The holographic rail isdisplayed as applied to a real-world stairway. The holographic rail isgenerated by the HMD device 20 matching parameters of the capturedschematic of FIG. 7.

FIG. 10A is an image of a holographic user interface for selecting styleoptions on a holographic structure where the holographic structure is ina first style configuration. A holographic icon 48 includes userinterface controls that enable a user to adjust the style or shape ofthe holographic structure 46 to preference. The figure displays acurrent holographic option 50 as chosen on the holographic userinterface 48. In this case, the option is a series of vertical cabinets.In use, a user may tap a finger in the air near the location of theholographic user interface 48 to select alternate options within theuser interface. Options include alternate styles, colors, materialsappearance, and positioning of the holographic structure 46.

FIG. 10B is an image of a holographic in a second style configurationafter receiving a selection in the holographic user interface. Thefigure displays a newly selected current holographic option 50 after theuser engaged with the holographic icon 48. In this case, the option wasof two vertical cabinets and two horizontal shelves.

By using the HMD device, and through the use of hand gestures, voicecommands and/or controlled gaze, the user can create and manipulatevarious 3D holographic (AR) objects, such as spheres and cubes. Forexample, the user can create and delete holographic objects, move androtate holographic structures 46, change colors, fill patterns, surfacetextures and decorations of holographic objects, etc. As shown, theseobjects are overlaid on the user's view of the real-world. Of course,many different variations of the above-described approaches arepossible.

FIG. 11 shows an example of various functional components of the HMDdevice 20, according to some embodiments. In FIG. 11, the functionalcomponents of the HMD device 20 include one or more instance of each ofthe following: a main processor 121, memory 122, transparent displaydevice 123, depth camera 124, head tracking cameras 125, video camera126, communication device 127, and audio subsystem 128, all coupledtogether (directly or indirectly) by an interconnect 129. Theinterconnect 129 may be or include one or more conductive traces, buses,point-to-point connections, controllers, adapters, wireless links and/orother conventional connection devices and/or media, at least some ofwhich may operate independently of each other.

The main processor(s) 121 individually and/or collectively control theoverall operation of the HMD device 20 and perform various dataprocessing functions. For example, the processor(s) 121 may provide orat least support the portable holographic user interface featuresdescribed above. Each processor 121 can be or include, for example, oneor more general-purpose programmable microprocessors, digital signalprocessors (DSPs), graphics processing unit (GPU), mobile applicationprocessors, microcontrollers, application specific integrated circuits(ASICs), programmable gate arrays (PGAs), or the like, or a combinationof such devices.

Data and instructions (code) 130 that configure the processor(s) 121 toexecute aspects of the technique introduced here can be stored in theone or more memories 122. Each memory 122 can be or include one or morephysical storage devices, which may be in the form of random accessmemory (RAM), read-only memory (ROM) (which may be erasable andprogrammable), flash memory, miniature hard disk drive, conventionalhard disk drive, or other suitable type of storage device, or acombination of such devices.

The depth camera(s) 124 can apply time-of-flight principles, forexample, to determine distances to nearby objects. The distanceinformation acquired by the depth camera 124 is used (e.g., byprocessor(s) 121) to construct a 3D mesh model of the surfaces in theuser's environment. The head tracking camera(s) 125 enable the HMDdevice 20 to continuously track the current location and orientation ofthe user's head by acquiring images of the user's real-worldenvironment. At least some of the functionality associated with surfacedetection and head tracking may be performed by the processor(s) 121.

The communication device(s) 127 enable the HMD device 20 to receive dataand/or commands from, and send data and/or commands to an externalprocessing system, such as a personal computer or game console, althoughin at least some embodiments the HMD device 20 can operate as astandalone device. Each communication device 127 can be or include, forexample, a universal serial bus (USB) adapter, Wi-Fi transceiver,Bluetooth or Bluetooth Low Energy (BLE) transceiver, Ethernet adapter,cable modem, DSL modem, cellular transceiver (e.g., 3G, LTE/4G or 5G),baseband processor, or the like, or a combination thereof. The audiosubsystem 128 includes at least one speaker and audio processingcircuitry to output sound effects to the user.

The machine-implemented operations described above can be implemented byprogrammable circuitry programmed/configured by software and/orfirmware, or entirely by special-purpose circuitry, or by a combinationof such forms. Such special-purpose circuitry (if any) can be in theform of, for example, one or more application-specific integratedcircuits (ASICs), programmable logic devices (PLDs), field-programmablegate arrays (FPGAs), system-on-a-chip systems (SOCs), etc.

Software to implement the techniques introduced here may be stored on anon-transitory machine-readable storage medium and may be executed byone or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable medium,” as the term is usedherein, includes any mechanism that can store information in a formaccessible by a machine (a machine may be, for example, a computer,network device, cellular phone, personal digital assistant (PDA),manufacturing tool, any device with one or more processors, etc.). Forexample, a machine-accessible medium includes recordable/non-recordablemedia (e.g., read-only memory (ROM); random access memory (RAM);magnetic disk storage media; optical storage media; flash memorydevices; etc.), etc.

EXAMPLES OF CERTAIN EMBODIMENTS

Certain embodiments of the technology introduced herein are summarizedin the following numbered examples:

1. A method comprising: receiving user input at a head mounted display(HMD) device; in response to the user input, capturing, by the HMDdevice, a position and an orientation of a physical tool that has beenpositioned relative to a real-world structure for which a schematicdiagram is to be created; and generating, by the HMD device, a firstvertex for a virtual schematic diagram of the real-world structure basedon the captured position and orientation of the tool, the virtualschematic diagram including a plurality of vertices positioned invirtual space, the plurality of vertices corresponding to points on thereal-world structure.

2. The method of example 1, further comprising: capturing, by the HMDdevice, additional positions and orientations of the tool, in responseto additional user inputs; generating, by the HMD device, the pluralityof vertices based on the additional user inputs; and generating, by theHMD device, the virtual schematic diagram from the plurality ofvertices.

3. The method of any of examples 1 to 2, wherein the virtual schematicdiagram further comprises measurements of distances between neighboringvertices.

4. The method of any of examples 1 to 3, wherein said receiving userinput further includes: capturing, by a camera on the HMD device, theposition and the orientation of the tool in response to aperspective-n-point user input.

5. The method of any of examples 1 to 4, wherein the HMD device includesa profile of a physical geometry of the tool, said capturing stepfurther comprising: computing the location and the orientation of thetool based on the profile.

6. The method of any of examples 1 to 5, wherein the HMD device includesstored data representing a profile of fiducial markers on the tool.

7. The method of any of examples 1 to 6, further comprising:identifying, by the HMD device based on the location and orientation ofthe peripheral tool, whether a corner of the peripheral tool, an edge ofthe peripheral tool, or a surface of the tool is within a specifiedproximity of the real-world structure; and wherein said generatingincludes determining, based on said identification, whether to generatea single vertex, a line, or a plane of the virtual schematic diagram.

8. The method of any of examples 1 to 7, further comprising: displaying,by the HMD device to a user of the HMD device, a 3D image of a secondstructure capable of being installed or constructed at the real-worldstructure, superimposed on the user's view of the real-world structurethrough the HMD device.

9. The method of any of examples 1 to 8, wherein said receiving userinput further includes: receiving, by the HMD device, an audible commandto capture the position and the orientation currently occupied by thetool.

10. The method of any of examples 1 to 9, wherein said generating thefirst vertex further comprises: basing the first vertex of the virtualschematic diagram on a first orientation of the tool, the firstorientation being such that the tool is in contact with the real-worldstructure at only a pointed portion of the tool, the first vertexlocated at a corresponding first location in digital space indicated bythe pointed portion of the tool.

11. The method of any of examples 1 to 10, wherein said generating thefirst vertex step further comprises: generating a line of the virtualschematic diagram bounded by the first vertex and a second vertex basedon a first orientation of the tool, the first orientation being suchthat the tool is in contact with an edge portion of the tool with thereal-world structure, the line generated at a first locationcorresponding to the edge portion of the tool and parallel to said edgeportion, the first vertex and second vertex positioned at a secondlocation and a third location respectively and determined by anenvironment mapping camera of the HMD device based upon recognizedangular changes in the real-world structure.

12. The method of any of examples 1 to 11, further comprising:generating a plane of the virtual schematic bounded by a plurality ofbounding vertices based on a first orientation of the tool, the firstorientation being a flat surface portion of the tool laid on thereal-world structure, the plane generated at a first locationcorresponding to the flat surface portion of the tool and correspondinglocations of the plurality of bounding vertices determined by anenvironment mapping camera of the HMD device based upon recognizedangular changes in the real-world structure.

13. A system comprising: a head mounted camera configured to receiveuser input including a tool and a real world structure; a processorconfigured to identify from the user input a plurality of positions anda corresponding plurality of orientations of the tool relative to thereal-world structure to generate a plurality of vertices positioned invirtual space, vertices the vertices corresponding to points on thereal-world structure; and a near-eye display configured to display avirtual schematic diagram of the real-world structure.

4 The system of example 13, further comprising: a memory configured tostore a profile of a physical geometry and design of the tool, theprofile used to compute the location and the orientation of the tool.

15. The system of any of examples 13 to 14, wherein the near-eye displayis further configured to display to a user a 3D image of a secondstructure capable of being installed or constructed at the real-worldstructure, superimposed on the user's view of the real-world structurethrough the HMD device.

16. The system of any of examples 13 to 15, further comprising: amicrophone coupled to the head mounted camera and configured to receiveaudible commands to capture the plurality of positions and thecorresponding plurality of orientations occupied by the tool.

17. A method comprising: acquiring, by a head mounted display (HMD)device, a virtual schematic diagram of a first physical structure basedon a user's use of a hand tool; and displaying, by the HMD device to auser of the HMD device, a 3D image of a second structure capable ofbeing installed or constructed at the real-world structure, superimposedon the user's view of the real-world structure through the HMD device.

18. The method of example 17, further comprising: receiving, by the HMDdevice, user input interacting with the 3D image of the secondstructure; and displaying, by the HMD device to the user of the HMDdevice, an amended configuration of the 3D image of the secondstructure.

19. The method of any of examples 17 to 18, further comprising:displaying, by the HMD device to the user of the HMD device, aholographic user interface including a plurality of style options forthe second structure; and receiving, by the HMD device, user inputinteracting with holographic user interface.

20. The method of any of examples 17 to 19, further comprising: storinga profile of a physical geometry of the hand tool on the HMD device; andcharacterizing the use of the hand tool based on the profile.

21 The method of any of examples 17 to 20, wherein the HMD deviceincludes stored data representing a profile of fiducial markers printedon the hand tool.

22. The method of any of examples 17 to 21, further comprising:identifying, by the HMD device from a location and an orientation of thehand tool, whether a corner of the hand tool, an edge of the hand tool,or a surface of the hand tool is within a specified proximity of thereal-world structure; and generating based on said identification, asingle vertex, a line, or a plane of the virtual schematic diagram.

23. A method comprising: accessing, by a mixed reality display device, avirtual schematic diagram of a real-world structure; generating, by themixed reality display device, image data representing a 3D image of asecond structure capable of being installed or constructed at thereal-world structure; and displaying, by the mixed reality displaydevice to a user of the mixed reality display device, the 3D image ofthe second structure, based on the virtual schematic diagram, such thatthe 3D image of the second structure appears to the user as insertedinto the user's view of the real-world structure while the user viewsthe real-world structure through the mixed reality display device.

24. The method of example 23, further comprising: positions of verticesof the virtual schematic diagram are determined based on detecting, bythe mixed reality device, positioning and orientation of a tool.

25. The method of any of examples 23 to 24, further comprising:positions of vertices of the virtual schematic diagram are determinedbased on detecting, by the mixed reality device, depth of surroundingspace including the real-world structure.

26. The method of any of examples 23 to 25, further comprising:receiving, by the mixed reality display device, user input interactingwith the 3D image of the second structure; and displaying, by the mixedreality display device to the user of the mixed reality display device,an amended configuration of the 3D image of the second structure.

27. The method of any of examples 23 to 26, further comprising:displaying, by the mixed reality display device to the user of the mixedreality display device, a holographic user interface including aplurality of style options for the second structure; and receiving, bythe mixed reality display device, user input interacting withholographic user interface.

28. The method of any of examples 23 to 27, further comprising:capturing, by the mixed reality display device, the virtual schematicdiagram of the real-world structure.

29. The method of any of examples 23 to 28, wherein said capturingfurther comprising: identifying, by the mixed reality display device, aposition and an orientation of a hand tool for each of a plurality ofvertices corresponding to the virtual schematic diagram.

30. The method of any of examples 23 to 29, wherein said capturing isperformed by a depth camera on the mixed reality display device.

Although the subject matter has been described in language specific tostructural features and/or acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as examples of implementing theclaims and other equivalent features and acts are intended to be withinthe scope of the claims.

The invention claimed is:
 1. A method by a mixed reality display device,comprising: accessing a virtual schematic diagram of a real-worldstructure; identifying a position and an orientation of a tool based onidentifying at least a subset of a plurality of fiduciary markers of thetool; generating a first vertex for the virtual schematic diagram of thereal-world structure based on the position and the orientation of thetool identified via the at least the subset of the plurality offiduciary markers; generating image data representing a 3D image of asecond structure capable of being installed or constructed at thereal-world structure; and displaying, to a user of the mixed realitydisplay device, the 3D image of the second structure, based on thevirtual schematic diagram, such that the 3D image of the secondstructure appears to the user as inserted into a user view of thereal-world structure while the user views the real-world structurethrough the mixed reality display device.
 2. The method of claim 1,wherein generating the first vertex further comprises generating a lineof the virtual schematic diagram bounded by the first vertex and asecond vertex based on a first orientation of the tool, the firstorientation being such that the tool is in contact with an edge portionof the tool with the real-world structure, the line generated at a firstlocation corresponding to the edge portion of the tool and parallel tosaid edge portion, the first vertex and second vertex positioned at asecond location and a third location respectively and determined basedupon recognized angular changes in the real-world structure.
 3. Themethod of claim 1, wherein generating the first vertex further comprisesgenerating a plane of the virtual schematic diagram bounded by aplurality of bounding vertices based on a first orientation of the tool,the first orientation being a flat surface portion of the tool laid onthe real-world structure, the plane generated at a first locationcorresponding to the flat surface portion of the tool and correspondinglocations of the plurality of bounding vertices determined based uponrecognized angular changes in the real-world structure.
 4. The method ofclaim 1, further comprising detecting a depth of surrounding spaceincluding the real-world structure to determine positions of vertices ofthe virtual schematic diagram.
 5. The method of claim 1, furthercomprising: receiving user input interacting with the 3D image of thesecond structure; and displaying, to the user of the mixed realitydisplay device, an amended configuration of the 3D image of the secondstructure.
 6. The method of claim 1, further comprising: displaying, tothe user of the mixed reality display device, a holographic userinterface including a plurality of style options for the secondstructure; and receiving user input interacting with the holographicuser interface.
 7. The method of claim 1, further comprising capturingthe virtual schematic diagram of the real-world structure.
 8. Ahead-mounted display (HMD) apparatus, comprising: a processor configuredto: access a virtual schematic diagram of a real-world structure;identify a position and an orientation of a tool based on identifying atleast a subset of a plurality of fiduciary markers of the tool; generatea first vertex for the virtual schematic diagram of the real-worldstructure based on the position and the orientation of the toolidentified via the at least the subset of the plurality of fiduciarymarkers; and generate image data representing a 3D image of a secondstructure capable of being installed or constructed at the real-worldstructure; and a display device configured to display, to a user of theHMD apparatus, the 3D image of the second structure, based on thevirtual schematic diagram, such that the 3D image of the secondstructure appears to the user as inserted into a user view of thereal-world structure while the user views the real-world structurethrough the HMD apparatus.
 9. The HMD apparatus of claim 8, wherein togenerate the first vertex the processor is further configured togenerate a line of the virtual schematic diagram bounded by the firstvertex and a second vertex based on a first orientation of the tool, thefirst orientation being such that the tool is in contact with an edgeportion of the tool with the real-world structure, the line generated ata first location corresponding to the edge portion of the tool andparallel to said edge portion, the first vertex and second vertexpositioned at a second location and a third location respectively anddetermined based upon recognized angular changes in the real-worldstructure.
 10. The HMD apparatus of claim 8, wherein to generate thefirst vertex the processor is further configured to generate a plane ofthe virtual schematic diagram bounded by a plurality of boundingvertices based on a first orientation of the tool, the first orientationbeing a flat surface portion of the tool laid on the real-worldstructure, the plane generated at a first location corresponding to theflat surface portion of the tool and corresponding locations of theplurality of bounding vertices determined based upon recognized angularchanges in the real-world structure.
 11. The HMD apparatus of claim 8,further comprising a depth camera configured to detect a depth ofsurrounding space including the real-world structure to determinepositions of vertices of the virtual schematic diagram.
 12. The HMDapparatus of claim 8, wherein: the processor is further configured toreceive user input interacting with the 3D image of the secondstructure; and the display device is further configured to display, tothe user of the HMD apparatus, an amended configuration of the 3D imageof the second structure.
 13. The HMD apparatus of claim 8, wherein: thedisplay device is further configured to display, to the user of the HMDapparatus, a holographic user interface including a plurality of styleoptions for the second structure; and the processor is furtherconfigured to receive user input interacting with the holographic userinterface.
 14. The HMD apparatus of claim 8, further comprising a cameraconfigured to capture the virtual schematic diagram of the real-worldstructure.
 15. A non-transitory computer readable medium comprisinginstructions that, when executed by a processor of a mixed realitydisplay device, cause the processor to: access a virtual schematicdiagram of a real-world structure; identify a position and anorientation of a tool based on identifying at least a subset of aplurality of fiduciary markers of the tool; generate a first vertex forthe virtual schematic diagram of the real-world structure based on theposition and the orientation of the tool identified via the at least thesubset of the plurality of fiduciary markers; generate image datarepresenting a 3D image of a second structure capable of being installedor constructed at the real-world structure; and display, to a user ofthe mixed reality display device, the 3D image of the second structure,based on the virtual schematic diagram, such that the 3D image of thesecond structure appears to the user as inserted into a user view of thereal-world structure while the user views the real-world structurethrough the mixed reality display device.
 16. The non-transitorycomputer readable medium of claim 15, wherein to generate the firstvertex comprises to generate a line of the virtual schematic diagrambounded by the first vertex and a second vertex based on a firstorientation of the tool, the first orientation being such that the toolis in contact with an edge portion of the tool with the real-worldstructure, the line generated at a first location corresponding to theedge portion of the tool and parallel to said edge portion, the firstvertex and second vertex positioned at a second location and a thirdlocation respectively and determined based upon recognized angularchanges in the real-world structure.
 17. The non-transitory computerreadable medium of claim 15, wherein to generate the first vertexcomprises to generate a plane of the virtual schematic diagram boundedby a plurality of bounding vertices based on a first orientation of thetool, the first orientation being a flat surface portion of the toollaid on the real-world structure, the plane generated at a firstlocation corresponding to the flat surface portion of the tool andcorresponding locations of the plurality of bounding vertices determinedbased upon recognized angular changes in the real-world structure. 18.The non-transitory computer readable medium of claim 15, furthercomprising instructions that, when executed by the processor, cause theprocessor to cause a depth camera to detect a depth of surrounding spaceincluding the real-world structure to determine positions of vertices ofthe virtual schematic diagram.